Skip to content

Expose provider token usage on responses - #151

Open
KotlinFactory wants to merge 3 commits into
huggingface:mainfrom
SolbachLeads:pr/token-usage
Open

Expose provider token usage on responses#151
KotlinFactory wants to merge 3 commits into
huggingface:mainfrom
SolbachLeads:pr/token-usage

Conversation

@KotlinFactory

@KotlinFactory KotlinFactory commented Mar 26, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a LanguageModelUsage type for provider-reported token usage
  • expose usage on LanguageModelSession.Response and streaming snapshots
  • map provider usage metadata for OpenAI, Open Responses, Anthropic, Gemini, and Ollama
  • add focused tests for response and stream usage exposure

Testing

  • xcrun swift build
  • xcrun swift test --filter MockLanguageModelTests

Notes

  • this PR is scoped to reporting token usage only
  • the branch is based directly on huggingface/main

@KotlinFactory

KotlinFactory commented Mar 26, 2026

Copy link
Copy Markdown
Contributor Author

Background: Tracking LLM cost on a per-user or per-task basis.

@KotlinFactory

KotlinFactory commented Apr 2, 2026

Copy link
Copy Markdown
Contributor Author

@mattt Does this make sense to you?

@lukasmartinelli

lukasmartinelli commented Apr 30, 2026

Copy link
Copy Markdown

this would be great to have

@noorbhatia

noorbhatia commented May 1, 2026

Copy link
Copy Markdown
Contributor

@KotlinFactory any plans to support MLXLanguageModel also?

@KotlinFactory

KotlinFactory commented May 2, 2026

Copy link
Copy Markdown
Contributor Author

@noorbhatia Sure why not. But lets focus on getting this merged first.

@mattt

KotlinFactory and others added 2 commits June 11, 2026 22:21
@KotlinFactory @claude
Expose provider reasoning traces on responses
450e9c6 
Parses vLLM/OpenRouter chat-completions reasoning fields (reasoning_content,
reasoning) and Responses API reasoning output items (reasoning_text /
summary_text parts) into a new optional LanguageModelSession.Response.reasoning.
Reasoning fields are decode-only and never re-encoded into upstream messages.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@KotlinFactory
Handle Anthropic thinking content blocks
5b6ad81
@mattt

mattt commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Hi @KotlinFactory. Yes, this makes sense, and I'm sorry for leaving your question unanswered. Would you rebase onto main? Once that's done, I'll review and merge. MLX usage can be a follow-up.

@mattt

mattt commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

One more thing while you're rebasing: Apple added usage reporting to Foundation Models in iOS/macOS 27 as LanguageModelSession.Usage, exposed as Response.usage and ResponseStream.Snapshot.usage. It splits into input (totalTokenCount, cachedTokenCount) and output (totalTokenCount, reasoningTokenCount).

Could you shape this PR's type to match that (name and structure) rather than a separate LanguageModelUsage? That way it's a drop-in when we move to the 27 API instead of a rename. Providers that don't report cached or reasoning tokens can leave those at zero.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@KotlinFactory @lukasmartinelli @noorbhatia @mattt